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ABSTRACT 


Nowadays there are many systems develop for agricultural purposes and 
most system implemented on the use of non-destructive technique not only to 
classify but also to determine the fruit ripeness. However, most of the studies 
concentrates using single technique to assess the fruit ripeness. This paper 
presents the work on mango ripeness classification using hybrid technique. 


Hybrid stands for mix or combination between two different elements, 
thus this study combined two different technique that is image processing 
and odour sensing technique in a single system. Image processing technique 
are implemented using color image that is HSV image color method to 
determine the ripeness of fruit based on fruit peel skin through color changes 
upon ripening. Whereas, odour sensing technique are implemented using 
Image PLOkeoeie sensors array to determine the fruit ripeness through smell changes upon 
Odour sensing ripening. The “Harumanis” and “Sala” mango was used for sample collection 
SVM based on two different harvesting condition that is unripe and ripe were 
evaluated using the image processing and followed by the odour sensor. 
Support Vector Machine (SVM) is applied as classifier for training and 
testing based on the data collected from both techniques. The finding shows 
around 94.69% correct classification using hybrid technique of image 
processing and odour sensing in a single system. 
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1, INTRODUCTION 

Fruits are consumed in Malaysia and globally whether it is eaten raw, cooked or consumed after 
being processed. It is because fruits guarantee health benefits such as energy supplier, disease prevention, 
healthy diets and well-built body in the long run. Fruits benefit the body immensely as they are natural 
sources of vitamins and minerals, which are essential for the proper functioning of the body, besides help to 
prevent from health complications such as cancer, high blood pressure, heat stroke, diabetes, etc. 
In Malaysia, fruits are divided either for fresh consumption usually for consumers and also production 
purposes that are develop by the industries [1]. Fresh consumption market where consumer will buy fresh 
fruits for daily intake or in cooking whereas production purposes market where the fruit are developed into 
edible purposes such as jams, sauces or marmalades but also for non-edible purposes, such as cosmetics, 
soaps, detergents, etc. The most important criteria when determine the fruits ripeness are the color of the 
skin, size and also odour where these three criteria are usually being classify by experts or farmers. However, 
the classification to determine the ripeness are often being evaluate differently by experts or farmers based on 
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own individual understanding. Therefore, with the aid of image processing and odour sensing technique, 
the ripeness classification resulting more accurate, reducing error and eliminate the manual ways. 

Digital image processing is vastly developed in many applications, thus its capabilities has attracted 
interest of several industries and chosen due to high consistency with accuracy [2]. Over recent years, 
this technology is ideal in the agricultural sector, where it can be used for quality assessment and 
classification based on size or maturity level of fruits [3]. Furthermore, using image processing can replaced 
the traditional technique that used to evaluate the fruit since ages because it uses non-destructive evaluation 
method. Odour sensing or odour gas sensor are also been recognized as hardware implementation for fruit 
ripeness classification. It is a fact that the smell or aroma can recognized the flavour, odour and volatile 
compounds inside the fruit. The odour sensor able to measure consistently because it can measure and 
identify quality factors such as impurities, taints and adulteration [4]. Besides that, odour sensor also have 
advantages that includes rapid, real-time volatiles detection and cheaper costs [4]. Therefore, this paper 
explores the implementation of image processing and odour sensing method together in a single system as a 
hybrid technique for mango ripeness classification. 

Based on the previous literature review by other researches, in [5], a study conducted to predict 
quality of “Alphonso” mangos using RGB images. Hierarchical clustering method was used to classify 
ripening period based on physico-chemical, color and textural parameters. Normalized differential index and 
area features models resulted 96.3% of correct prediction that are extracted from the images. Besides that, 
[6] proposed an approach using mean pixel values in RGB and HSB color spaces. The “Chokanan” mango 
was selected and a model was develop based on hue from the HSB using linear regression analysis with 
average success rate of 95.67%. Authors [7] implemented system that detect “Gincu” mango ripening 
through color using C language, computer vision and artificial neural network. The mango fruit sortation 
classifies with 94% of accuracy and also conclude that speed and accuracy are important when handling with 
hidden layer using neural network. 

On the other hand, there are also previous research that implemented odour sensing towards mango 
fruit for ripeness classification. [4] implemented an array of semi-conductor gas sensors to determine the 
pattern of “Harumanis” mango ripeness. Each stages of ripeness are identified and trained using artificial 
neural network where this method proven that the sensor array able to leaves a characteristic pattern for 
different stages of ripeness. Authors [8] also using semi-conductors gas sensors using Principal Component 
Analysis (PCA) and also Artificial Neural Network (ANN) for ripeness assessment for “Harumanis” mango. 
It also concluded that the particular method able to leaves significant characteristic pattern through different 
ripeness level. In [9] can also determine the “Chokanan” mango ripeness using zNoseTM electronic nose that 
used electronic detector with basis of (gas) chromatography with system controller. The system potentially 
able to predict mango fruit maturity during harvesting at the optimal harvest time. Therefore, based on the 
previous studies, color and smell are important parameter to identify the mango ripeness, thus by 
implementing image processing and odour sensing together as an automated system increases the reliability 
and accuracy for mango ripeness classification. 


2. RESEARCH METHOD 
2.1. Image Processing Technique 
2.1.1 Fruit Preparation 

Mango images are used for fruit ripeness based on the fruit peel color changes. “Harumanis” and 
“Sala” mangos are selected as it differs in term of color changing upon ripening. The mangos skin differs 
from unripe to ripe, for example each level of ripeness based on the mangos skin sometimes can be 
distinguished by naked eye. Unripe can be determine when the skin is greenish and will completely or 
slightly changing to yellowish color upon ripening. Sometimes there are also mangos that change from darker 
green to lighter green when it is ripe but there also no color changes upon ripening for some mangos but 
other properties enable to identify its ripeness such as odour. Figure | illustrates the “Harumanis” and “Sala” 
mango images that has different tone color skin correlations to identify whether it 1s unripe or ripen. 
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Figure 1. Different type of mangos where (a) and (b) 1s the “Sala” mango whereas (c) 
and (d) is the “Harumanis” mango 


2.1.2 Sample Collection 

The data set of “Harumanis” and “Sala” mango samples with total of 228 images each respectively 
used for image processing. For image processing technique, these fruit images are divided into training and 
testing set, where 30% of the fruit images from each group are used to train the technique and remaining 
images serve as the testing set. The fruit samples are manually harvested from Bukit Temiang, Perlis to 
ensure randomness in fruit collection process. The samples are also being proved by the experts in the 
relevant field for manual prediction of maturity so that it can be compared to the techniques that are used in 
the system. The fruit data are captured using Logitech720p Webcam C905 shown in Figure 2 and saved at 
the database consists of all the images that are separated by different folder according to its categories so that 
it is easy to retrieve. 





Figure 2. Fruit captured for data acquisition using Logitech720p Webcam C905 


2.1.3 Pre-Processing 

Pre-processing is a useful and important step in order to increase overall accuracy of image 
processing technique and the target image should same attribute. It is also a technique used for eliminating 
insignificant information based on data collection of certain pixels and therefore acquiring the images are 
important to ensure all variable are controlled to make sure that the data are valid. In pre-processing, it is 
necessary that all images being are in similar category so that the images can processed accurately. 
The images undergone image pre-processing and for this case, the images are pre-processed when the photo 
are taken by using white background when the image are captured. White background is chosen as the fruit 
background as the controlled parameter because it can capture image clearer and eliminate excess lighting 
from the surroundings. 


2.1.4 RGB to HSV Conversion 

The sample collected are in form of the RGB spectrum that is in red, green and blue channel with 
each pixel in the channels having value ranging from O to 255 [3]. After that, the RGB will be convert into 
HSV color model where the image is separated into hue, saturation and value channel or image using set of 
equations from (1), (2), (3), (4), (5). In HSV, the value varies between O and 1 for each channel and Figure 3 
illustrate the difference when an original image converted to hue, saturation and value channel. 
The conversion from RGB to HSV 1s to detect the color presence in the images more accurately based on 
image acquired. The significant advantages of HSV than other color spaces is the compatibility that is similar 
to human intuition besides the possibility of using single, two or all color feature that allowing the 
implementation of much faster algorithms and also accurate result. 


(1) 


. = cos"*{ +[(R-G)+(R-B)| 
= 


J (R-G)*+(R—-B)(G—B) 
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H=H,,ifB<G (2) 

H = 360° — H,,if B>G (3) 
_ max(R,G,B)—min(R,G,B) 

a max(R,G,B) (4) 
__ max(R,G,B) 

= 255 (9) 
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Figure 3. Original image (RGB) converted to HSV color model 


Fruits can be categorized into several ripeness or maturity levels based on fruit skin color changes. 
For this implementation of fruit using HSV, mango images are used to test the accuracy and reliability of the 
methods. The fruit skin differs from unripe to ripe, for example each level of ripeness can be set apart based 
on the fruit skin color. Unripe can be determine when the skin is pale according to the fruit and will slightly 
or completely changed to another color upon ripening. Therefore, the HSV color space or model correspond 
closely to color perception that similar to human interpretation [3]. 


2.1.5 Thresholding 

In this stage, there are used of threshold method in order to isolate the colors by applying 
multiple masks. A low threshold and high threshold mask are implemented for hue, saturation and 
value images. Anything pixel within these thresholds will be set to 1 and the remaining pixels will be zero. 
In threshold method, the global threshold (level) played an important role in order to set a boundary of HSV 
grayscale value. In this concept, graythresh function in MATLAB which is represented as global image 
threshold using Otsu's method is applied to compute the threshold level, which can minimize the intraclass 
variance of the black and white pixel [10]. By using the fuction, the threshold masking levels are obtained 
and produced as binary output. 


2.1.6 Classification 

The data images that have been processed are analyzed to determine the fruit ripeness. Each pixel of 
image containing RGB and HSV color space data and each color component ranging of 0 to 255 for RGB 
and range of 0 to 1 for HSV. The system applied Support Vector Machine (SVM) learning machine for 
ripeness stages classification based on the image processing technique data. The inputs are the training 
dataset feature vectors that is the mean HSV value from the HSV conversion and their corresponding classes, 
whereas the outputs are the ripeness stage of the image in the testing dataset. SVM first transform the input 
data into a higher-dimensional space by using a kernel function and an optimal hyperplane is constructed 
between two classes whether it is unripe or ripe to separate the data in transformed space [11]. Specific range 
feature decides on to acquire specific result for example if the mean HSV is less or higher than the boundary 
that have been set, thus it can determine whether the mango is unripe or ripe. The inputs value trained using 
the SVM classifier and later test the accuracy. 


2.2. Odour Sensing Technique 
2.2.1 Odour Data Acquisition 

The mango fruits are also applied in odour sensing technique for ripeness classification. This odour 
sensing technique are not similar with image processing technique. Odour sensors can sense and record the 
fruit smell whether it is ripe or unripe because different fruit conditions or maturity gives different reading. 
There are several types of metal oxide gas sensors implemented and applied in this hardware development as 
stated in Table 1 [12]. 
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Table 1. List of Sensors Used for Sensor Array 


Label Sensors Odour Types 
Sl TGS813 Combustible Gases 
S2 TGS821 Hydrogen 
S3 TGS822 Ethanol 
S4 ME3-C2H4 Ethylene 
S5 TGS825 Hydrogen Sulphide 
S6 TGS826 Ammonia 
S7 TGS830 Chlorofluorocarbons 
S8 TGS2600 Air Contaminants 
S9 TGS2620 Alcohol & Solvent Vapours 


The hardware development which consists of the sensor arrays or Odour Gas Sensor (OGS) with 
addition of exhausted fan, temperature and humidity display are implemented together in the prototype. 
The initial sensor arrays prototype started with circuit design of sensors and before tested on the PCB board. 
The circuit can convert the changing in conductivity to output signal with corresponding to gas concentration. 
Based on the characteristics of the metal gas oxide, where the changing of resistance and voltage of sensors 
happen when it reacts between the negative charge of metal oxide surface and deoxidizing gas. The odour 
data based on the mangos sample reading was taken to differentiate the level of maturity from each batch of 
fruits one by one. The odour data from the OGS was obtained through the National Instrument (NI) Data 
Acquisition (DAQ) device, where it can read the volatilities changes from the sensor. The NI-DAQ device 
also recorded the odour data as a reference library for training and testing. Therefore, by using OGS it can 
differentiate the mango ripeness through smell properties and by implementing several gas sensors also will 
also increase the accuracy of fruit smell and aroma detection and also classification [12]. 


2.2.2 Odour Pre-Processing 

As for odour pre-processing it is slightly different than image pre-processing as the odour data must 
also be valid where all the variable has been controlled. There are issues when using metal gas oxide, 
which it had a high sensitivity to temperature and humidity. Therefore, the developed OGS are installed in 
the prototype together with the webcam camera that appear similar to a gas chamber box for data collection 
as shown in Figure 4. Two exhausted fans are also fixed at the prototype to ensure that after the odour 
reading collection, the excess odour inside the prototype are eliminate using the fan. Thus, the prototype 
surrounding is at initial state and the data collection for other samples will be valid and accurate. OGS will 
detect the mango odour, thus the output voltage from the sensor will be sent to NI-DAQ for data collection, 
processing and classification through the computer. 





Figure 4. Prototype that implemented image processing and OGS in a single system 


2.2.3 Odour Measurement Technique 

Measurement process for odour sensing technique are divided into three different part or phases that 
1s initialization, measuring, and purging. The measurement process starts by placing mangos in the prototype 
chamber for initializes phase. To strengthen the aroma or smell concentration and obtaining higher sensor 
responses, the sensors boot up to a steady state after 200s. After that, the measurement phase starts where 
sensors detected the mango smell and reading through NI-DAQ device for around 300s, which enough for 
sensors obtained stable value. Lastly the purging phase will occur after the measurement complete. The main 
purpose of this phase is to remove the excess odour by using surrounding air through the fan so that the 
sensors could go to the initial baseline values [13]. After the excess odour existed in the prototype chamber is 
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exhausted by the fan, new measurement and reading started again. The collected reading was conducted at 
the temperature around 27°C to 33°C and maintained constant. The data value was collected from the odour 
gas sensor (OGS) using the NI DAQ device and NI LABVIEW software is used to display the waveform 
chart consisting of each sensor data through the Graphical User Interface (GUI) inside NI LABVIEW 
software. 


2.2.4 Classification 

SVM learning machine as classifier are also used for odour sensing training and testing, where it 1s 
useful to understand which combinations of pairs of sensors are gives the able to differentiate between ripe 
and unripe smell. By testing the SVM with untrained data, the classifier 1s able to detect the maturity level of 
mangos using OGS technique. System started and test continuously and the mean value are calculated for 
each sensor output to feed into the SVM for training. Using SVM, the method performs without waiting for 
the sensor to fully reach up to a constant value whereby a quick detection method of fruit maturity 1s detected 
in a few seconds based on the rising signal of the odour sensor. 


2.3. Image and Odour Combination (Hybrid Technique) 

As mentioned earlier, hybrid stands for combination between two different elements and this study 
implemented software and hardware combination by using image processing technique and also OGS 
technique. The approach is to put the both techniques side-by-side and focus on their interaction in a 
problem-solving task. The conceptual view allows one to abstract from the individual techniques as well as to 
study the individual contribution of each component. The benefits of integrated techniques include 
robustness, improved performance and increased problem-solving capabilities for classification [14]. 
Figure 5 illustrated the image processing and OGS using hybrid technique for fruit ripeness classification. 
Hybrid systems have limitations caused by the increased complexity therefore, both of the technique is 
integrated together in LabVIEW software that also acts as GUI for simpler solution. The image processing 
process and also SVM classifier that are employed through MATLAB software are integrated with the OGS 
hardware, thus the LabVIEW acts as the integrated medium for the fruit ripeness classification decision in a 
single system. 





(b) 


Figure 5. Prototype that implemented image processing (a) and OGS (b) 
in a single system 


3. RESULTS AND ANALYSIS 

After the fruit data has been collected and processed through image processing along with odour 
sensing, the data are separated whereby 30% used for training and the remaining used for testing. The reason 
using this ratio is because to decrease the time for samples training and increase the accuracy result through 
increment of the testing samples. Table 2 tabulated the samples used for training and testing with two 
different classes that 1s unripe and also ripe, besides the SVM learning machine are used for training and 
testing so that the system can classify the mango ripeness. 


Table 2. Samples Used for Training and Testing 
No. of Fruit Samples Used for 


Fruits Total Fruit Samples Training Testing 
Unripe Ripe —_Unripe Ripe 
Harumanis mango 228 34 34 80 80 
Sala mango 228 34 34 80 80 
Total 456 68 68 160 160 
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To ensure that the system are operating and robust, the accuracy of the system for mango ripeness 
classification are calculated using (6). 


TP+TN 


Accuracy = — 
TP +FN+FP+TN (6) 


Where : TP = true positive 

TN = true negative 

FP = false positive 

FN = false negative 

Figure 6 shows the mean HSV value corresponding to the mango images according to the respective 
levels of ripeness. Based on the result, there are difference for unripe and ripe level can be seen in each H, 
S and V value. The mean H & V differs slightly around its value area whereas, mean V plays the crucial 
value to determine the mean HSV and also the ripeness condition where the unripe level are around 0.6 and 
below whereas ripe level delivers 0.6 and above. The graph also shows that the hue reading linearly increase 
when the the saturation and value channel reading decrease based on the images running in the program. 
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Figure 6. Mean value for ripeness level using HSV 


The result from Figure 7 shows the histogram data for the test image classification through the SVM 
technique. Based on the histogram, the hue value is not entirely differing for unripe or ripe images, but for 
the saturation and value it is not similar. It can be seen that the saturation peak for the ripe fuit is higher than 
the unripe fruit. Whereas for the value histogram, the ripe fruit peak value is near to the threshold, while for 
the unripe fruit the peak is slightly further away from the threshold value that is 0.6. Therefore, the HSV 
image color basically with addition of thresholding and SVM as classifier can determine the ripeness for 
mango fruit images and theoretically it correspond similar to the color intepretation by human. 
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Figure 7. Histogram analysis graph for (a) unripe fruit (b) ripe fruit 
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Table 3 summarizes the classification result using the image processing technique proved that the 
technique able to achieve high precision of 91.88% of accuracy in determine fruit ripeness. Based on the 
result, it can classify and determine all of the 456 images of mangos. There are some misclassify samples due 
to some mango images that contained more greenish color when it ripe and also some yellowish color when it 
is actually still unripe. Nevertheless, the overall performance by using image processing to determine the 
fruit classification is good and the technique is suitable for certain fruit that changed color from unripe to ripe 
such as mango 


Table 3. Classification of Mango Images using Image Processing Technique 


For Image 
Unripe Ripe Acc 
TP FN TN FP 
Harumanis mango 76 4 74 6 93.75 
Sala mango 74 6 70 10 90.00 
Total 150 10 144 16 91.88 


The sensors that indicates the difference of fruit ripeness are shown in Figure 8 where the chart 
illustrates the sensor reading between unripe and ripe. Pairs of sensors which made the best discrimination 
between unripe and ripe class are sensors S3 (ethanol) and S4 (ethylene) that provided the huge difference 
reading of sensors than other sensors for discriminating the ripening stages. 
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Figure 8. Sensor array reading using OGS for unripe and ripe mangos 


Table 4 sum up the classification result using odour sensing technique which shows the odour gas 
sensor array able to achieve high precision of 97.50% of accuracy in determine fruit ripeness. Similar amount 
of data that 1s 456 used to match the images data and hence, it can classify and determine all of the 456 odour 
data of mangos with few are misclassify maybe because some reading recorded stronger smell although it is 
still unripe. The overall performance by using odour sensing to determine the fruit classification is suitable 
and technique is fit for fruit that that has high concentration of smell or aroma from unripe to ripe stage that 
are similar as mango. 


Table 4. Classification of Mango Images using Odour Sensing Technique 


For Odour 
Unripe Ripe Acc 
TP FN TN FP 
Harumanis mango 78 2 80 0 98.75 
Sala mango 74 6 80 0 96.25 
Total 152 8 160 0 97.50 
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As a result, the overall accuracy of fruit ripeness classification through image processing and odour 
sensing as a hybrid system achieved accuracy of 94.69% by combining both accuracy that is 91.88% and also 
97.50%. The main point is that the hybrid technique based on image processing and odour sensing can 
accommodate each other. This means that if the mango ripeness cannot identify accurately using image 
processing, it can be identified through odour sensing and vice versa. Hence, the system can be implemented 
for other fruits because has the ability to cover wider range of fruits that have either color or smell properties 
upon ripening. 


4. CONCLUSION 

In this paper, hybrid technique as a combination method used for mango ripeness classification by 
implementing image processing and odour sensing technique. The hybrid approach is to put the both 
techniques side-by-side and can accommodate each other in case the fruit cannot be determine using single 
technique. HSV color models and array gas sensors used as basic development to extract the required 
information to classify mango ripeness. The data obtained for the classification shows that both the unripe 
group and ripe group can be achieved and identified with an accuracy of 94.69%. As a conclusion, the image 
processing technique combined with odour sensing able to differentiate ripeness in mango and using the 
hybrid technique it can be implement for other types of fruits in the future. 
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